The datasetsData processingNetwork architectureSummary and Conclusionswhat is new?- the use of the transformer(the same that powers ChatGPT and all recent AI models)- integrating it with fMRI data in a series of taskshow did they do it?transformermodelagegendershizofreniameanvoxelstdevvoxelglobalnormalizationnumber of voxelsmeanvoxel22number of voxelsstdevvoxelvoxelnormalizationmeanvolumestdevvolumemeanvolumenumber of volumes22number of volumesstdevvolumez-scoresglobalnormalizationEncoderDecoderz-scoresglobalnormalizationpositional encodingtoken type embeddingadding the CLS tokenlayer normalizationdropoutGELU(linear(x))hidden_statenormalization(dropout(linear(hidden_state))+x)xxlast_hidden_statepooler_outputthis is the processed CLS tokenin the fmri paperthey go to bottleneck out0122928272639263826372102640 elementshidden_sizemax_position_embeddingslinearlinearlinearQKVQKTsoftmaxdkVlineardropoutlayer norm+linearGELUlineardropoutlayer normSelf attention2640->3072+3072->2640BERT layerpositionalembeddingBERTlayerBERTlayer21x264021x2640first one is CLSfirst one isprocessed CLSBidirectionalEncoderRepresentations fromTransformersEncoderz-scoresglobalnormalizationbottleneck inQUESTIONSthank youBERT Key Innovations Novel application of transformers to fMRI data analysis Three-phase training approach: autoencoder, transformer pre-training, and task-specific fine-tuning Effective combination of CNNs and transformers for spatiotemporal fMRI processing Major Findings State-of-the-art performance on multiple fMRI prediction tasks: Age prediction: L1 error of 2.73 years Gender classification: 94.09% accuracy Schizophrenia detection: Up to 88.2% accuracy (CNP dataset) Strengths Versatility across different prediction tasks and datasets Ability to capture both spatial and temporal patterns in fMRI data Effective use of self-supervised pre-training on large fMRI datasets Limitations and Future Directions Limited exploration of sequence length and stride parameters Potential for further optimization of model architecture Opportunity for more extensive ablation studies Implications New possibilities for advanced fMRI analysis in neuroscience and clinical applications Potential for improved understanding of brain function and neurological disorders Framework for applying transformer models to other types of medical imaging data Conclusion TFF demonstrates the power of adapting advanced AI techniques to neuroimaging, opening new avenues for brain research and clinical diagnostics.Phase 1: autoencoder pre-trainingPhase 2: transformer pre-trainingPhase 3: fine-tunningage?gender?shizofrenia?z-scoresbottleneck inbottleneck outz-scoresglobalnormalizationL1 lossL1 lossMSE lossVGGnetworktop 10%of activatedvoxelsVGGnetworkEncoderconv3dkernel=3padding=1stride=12ch4ch8ch16chdown block 1down block 232chdown block 3final blockdropoutdim_0dim_1dim_2dim_3depthdepthx 2depthx 4depthx 832chdim_3depthx 8batch×T,ch,W,H,Dgroupnormleakyreluconv3d3x3x3stride 1padding 1leakyreluconv3d3x3x3stride 1padding 1n_chn_chgroup_norm0relu0conv0group_norm1relu1conv2dropoutgrp:nch/4groupnormgrp:nch/4conv3dkernel 3padding 1n_ch2 × n_chstride 2groupnormleakyreluconv3d3x3x3stride 1padding 132ch2chdepthx 8depth/ 2grp:8BottleNeck inflattenreshapeorfullyconnected2640dim_3real ageBCE losslinearsigmoidgenderclassification(binary classification)healthy / schizofreniaBCE losslinearsigmdoipathologicalclassification(binary classification)linearleaky relureal ageageprediction(regression)L1 lossgroupnormleakyreluconv3d3x3x3stride 1padding 1leakyreluconv3d3x3x3stride 1padding 1n_chn_chgroup_norm0relu0conv0group_norm1relu1conv2dropoutgrp:nch/4groupnormgrp:nch/4unflattengroupnormgrp:22chdepth/ 2dim_3leakyreluconv3d3x3x3stride 1padding 132chdim_3depthx 82640BottleNeck outupgreen0upgreen1upgreen21chconv3d1x1x1stride 1padding 1output_blockDecoderclass Decoder(BaseModel):32chdim_3depthx 816chdim_2depthx 48chdim_1depthx 24chdepth4chdepthdim_0dim_0outChannelsdim_0conv3dkernel=3padding=1stride=1conv3d1x1x1stride 1padding 1nearestneighbourupsampletorch.nn.Upsample
1
  1. New frame
  2. New frame
  3. New frame
  4. New frame
  5. New frame
  6. New frame
  7. New frame
  8. New frame
  9. New frame
  10. New frame
  11. New frame
  12. New frame
  13. New frame
  14. New frame
  15. New frame
  16. New frame
  17. New frame
  18. New frame
  19. New frame
  20. New frame
  21. New frame
  22. New frame
  23. New frame
  24. New frame
  25. New frame
  26. New frame
  27. New frame
  28. New frame
  29. New frame
  30. New frame
  31. New frame
  32. New frame
  33. New frame
  34. New frame
  35. New frame
  36. New frame
  37. New frame
  38. New frame
  39. New frame
  40. New frame
  41. New frame
  42. New frame
  43. New frame
  44. New frame
  45. New frame
  46. New frame
  47. New frame
  48. New frame
  49. New frame
  50. New frame
  51. New frame
  52. New frame
  53. New frame
  54. New frame
  55. New frame
  56. New frame
  57. New frame
  58. New frame
  59. New frame
  60. New frame
  61. New frame
  62. New frame
  63. New frame
  64. New frame
  65. New frame
  66. New frame
  67. New frame
  68. New frame
  69. New frame
  70. New frame
  71. New frame
  72. New frame
  73. New frame
  74. New frame
  75. New frame
  76. New frame
  77. New frame
  78. New frame
  79. New frame
  80. New frame
  81. New frame
  82. New frame
  83. New frame
  84. New frame
  85. New frame